Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new db bucket for light client updates #5205

Merged
merged 1 commit into from
Feb 27, 2023

Conversation

wemeetagain
Copy link
Member

Motivation

Avoid bad UX error:
Error: LightClientUpdate size 25360 not equal fixed size 25368

Description

Use a new bucket for light client updates since #5027 changed what we store on disk.

@wemeetagain wemeetagain requested a review from a team as a code owner February 27, 2023 01:25
@twoeths
Copy link
Contributor

twoeths commented Feb 27, 2023

posting full stack trace for later reference

Feb-22 10:54:12.744[chain]           ^[[31merror^[[39m: Error updating best LightClientUpdate syncPeriod=615, slot=5046270, blockRoot=0x849a1f3cb9917bb744e7a5654e1ac88e8454fc3dc0473fb5d17124bf945b1dde LightClientUpdate size 25360 not equal fixed size 25368
Error: LightClientUpdate size 25360 not equal fixed size 25368
    at ContainerType.getFieldRanges (/usr/app/node_modules/@chainsafe/ssz/src/type/container.ts:399:15)
    at ContainerType.value_deserializeFromBytes (/usr/app/node_modules/@chainsafe/ssz/src/type/container.ts:209:30)
    at ContainerType.deserialize (/usr/app/node_modules/@chainsafe/ssz/src/type/abstract.ts:135:17)
    at BestLightClientUpdateRepository.decodeValue (file:///usr/app/packages/beacon-node/src/db/repositories/lightclientBestUpdate.ts:37:72)
    at BestLightClientUpdateRepository.get (file:///usr/app/packages/db/src/abstractRepository.ts:63:17)
    at LightClientServer.maybeStoreNewBestUpdate (file:///usr/app/packages/beacon-node/src/chain/lightClient/index.ts:555:28)
    at LightClientServer.onSyncAggregate (file:///usr/app/packages/beacon-node/src/chain/lightClient/index.ts:535:7)

@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: f744d7a Previous: caecd85 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 760.76 us/op 768.46 us/op 0.99
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 46.669 us/op 43.346 us/op 1.08
BLS verify - blst-native 1.2111 ms/op 1.1665 ms/op 1.04
BLS verifyMultipleSignatures 3 - blst-native 2.4613 ms/op 2.3728 ms/op 1.04
BLS verifyMultipleSignatures 8 - blst-native 5.2866 ms/op 5.0945 ms/op 1.04
BLS verifyMultipleSignatures 32 - blst-native 19.125 ms/op 18.410 ms/op 1.04
BLS aggregatePubkeys 32 - blst-native 25.431 us/op 24.635 us/op 1.03
BLS aggregatePubkeys 128 - blst-native 99.633 us/op 96.137 us/op 1.04
getAttestationsForBlock 54.525 ms/op 52.799 ms/op 1.03
isKnown best case - 1 super set check 256.00 ns/op 271.00 ns/op 0.94
isKnown normal case - 2 super set checks 260.00 ns/op 267.00 ns/op 0.97
isKnown worse case - 16 super set checks 241.00 ns/op 263.00 ns/op 0.92
CheckpointStateCache - add get delete 5.1620 us/op 5.0980 us/op 1.01
validate gossip signedAggregateAndProof - struct 2.7845 ms/op 2.7272 ms/op 1.02
validate gossip attestation - struct 1.3057 ms/op 1.2990 ms/op 1.01
pickEth1Vote - no votes 1.1831 ms/op 1.2222 ms/op 0.97
pickEth1Vote - max votes 10.280 ms/op 10.622 ms/op 0.97
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.8373 ms/op 8.7359 ms/op 1.01
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.459 ms/op 13.915 ms/op 1.04
pickEth1Vote - Eth1Data fastSerialize value x2048 625.46 us/op 611.34 us/op 1.02
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.3304 ms/op 8.0948 ms/op 0.91
bytes32 toHexString 482.00 ns/op 493.00 ns/op 0.98
bytes32 Buffer.toString(hex) 338.00 ns/op 347.00 ns/op 0.97
bytes32 Buffer.toString(hex) from Uint8Array 541.00 ns/op 545.00 ns/op 0.99
bytes32 Buffer.toString(hex) + 0x 336.00 ns/op 348.00 ns/op 0.97
Object access 1 prop 0.16600 ns/op 0.16400 ns/op 1.01
Map access 1 prop 0.15500 ns/op 0.16700 ns/op 0.93
Object get x1000 6.5840 ns/op 7.0580 ns/op 0.93
Map get x1000 0.61000 ns/op 0.58800 ns/op 1.04
Object set x1000 51.383 ns/op 50.389 ns/op 1.02
Map set x1000 42.368 ns/op 42.805 ns/op 0.99
Return object 10000 times 0.23420 ns/op 0.23240 ns/op 1.01
Throw Error 10000 times 4.1334 us/op 4.0632 us/op 1.02
fastMsgIdFn sha256 / 200 bytes 3.4500 us/op 3.3520 us/op 1.03
fastMsgIdFn h32 xxhash / 200 bytes 279.00 ns/op 282.00 ns/op 0.99
fastMsgIdFn h64 xxhash / 200 bytes 380.00 ns/op 384.00 ns/op 0.99
fastMsgIdFn sha256 / 1000 bytes 11.601 us/op 11.404 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 406.00 ns/op 404.00 ns/op 1.00
fastMsgIdFn h64 xxhash / 1000 bytes 449.00 ns/op 449.00 ns/op 1.00
fastMsgIdFn sha256 / 10000 bytes 103.52 us/op 102.70 us/op 1.01
fastMsgIdFn h32 xxhash / 10000 bytes 1.9080 us/op 1.8940 us/op 1.01
fastMsgIdFn h64 xxhash / 10000 bytes 1.3620 us/op 1.3450 us/op 1.01
enrSubnets - fastDeserialize 64 bits 1.2630 us/op 1.2490 us/op 1.01
enrSubnets - ssz BitVector 64 bits 489.00 ns/op 474.00 ns/op 1.03
enrSubnets - fastDeserialize 4 bits 168.00 ns/op 170.00 ns/op 0.99
enrSubnets - ssz BitVector 4 bits 489.00 ns/op 476.00 ns/op 1.03
prioritizePeers score -10:0 att 32-0.1 sync 2-0 93.848 us/op 93.548 us/op 1.00
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 119.71 us/op 114.98 us/op 1.04
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 167.99 us/op 162.05 us/op 1.04
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 301.64 us/op 291.65 us/op 1.03
prioritizePeers score 0:0 att 64-1 sync 4-1 358.57 us/op 359.41 us/op 1.00
array of 16000 items push then shift 1.6314 us/op 1.6197 us/op 1.01
LinkedList of 16000 items push then shift 8.8210 ns/op 8.6600 ns/op 1.02
array of 16000 items push then pop 77.081 ns/op 76.036 ns/op 1.01
LinkedList of 16000 items push then pop 8.5200 ns/op 8.4210 ns/op 1.01
array of 24000 items push then shift 2.3450 us/op 2.3028 us/op 1.02
LinkedList of 24000 items push then shift 8.7790 ns/op 8.5260 ns/op 1.03
array of 24000 items push then pop 75.439 ns/op 73.749 ns/op 1.02
LinkedList of 24000 items push then pop 8.4900 ns/op 8.2540 ns/op 1.03
intersect bitArray bitLen 8 13.248 ns/op 12.874 ns/op 1.03
intersect array and set length 8 77.214 ns/op 75.467 ns/op 1.02
intersect bitArray bitLen 128 44.012 ns/op 42.762 ns/op 1.03
intersect array and set length 128 1.0559 us/op 1.0229 us/op 1.03
Buffer.concat 32 items 2.9260 us/op 3.2620 us/op 0.90
Uint8Array.set 32 items 2.6760 us/op 2.6750 us/op 1.00
pass gossip attestations to forkchoice per slot 3.2845 ms/op 3.2318 ms/op 1.02
computeDeltas 2.9021 ms/op 2.7798 ms/op 1.04
computeProposerBoostScoreFromBalances 1.8059 ms/op 1.7517 ms/op 1.03
altair processAttestation - 250000 vs - 7PWei normalcase 2.1478 ms/op 2.0842 ms/op 1.03
altair processAttestation - 250000 vs - 7PWei worstcase 3.3070 ms/op 3.0989 ms/op 1.07
altair processAttestation - setStatus - 1/6 committees join 141.05 us/op 132.10 us/op 1.07
altair processAttestation - setStatus - 1/3 committees join 280.66 us/op 256.67 us/op 1.09
altair processAttestation - setStatus - 1/2 committees join 363.36 us/op 332.79 us/op 1.09
altair processAttestation - setStatus - 2/3 committees join 461.06 us/op 420.58 us/op 1.10
altair processAttestation - setStatus - 4/5 committees join 649.48 us/op 610.87 us/op 1.06
altair processAttestation - setStatus - 100% committees join 764.11 us/op 688.21 us/op 1.11
altair processBlock - 250000 vs - 7PWei normalcase 16.342 ms/op 17.995 ms/op 0.91
altair processBlock - 250000 vs - 7PWei normalcase hashState 30.268 ms/op 26.249 ms/op 1.15
altair processBlock - 250000 vs - 7PWei worstcase 50.432 ms/op 51.023 ms/op 0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState 71.288 ms/op 65.466 ms/op 1.09
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0049 ms/op 2.0103 ms/op 1.00
phase0 processBlock - 250000 vs - 7PWei worstcase 28.653 ms/op 26.537 ms/op 1.08
altair processEth1Data - 250000 vs - 7PWei normalcase 472.25 us/op 465.26 us/op 1.02
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 8.3250 us/op 6.7560 us/op 1.23
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 23.475 us/op 19.324 us/op 1.21
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 9.6340 us/op 8.5740 us/op 1.12
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 8.4040 us/op 6.8810 us/op 1.22
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 80.303 us/op 74.438 us/op 1.08
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 652.27 us/op 618.39 us/op 1.05
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 915.53 us/op 908.84 us/op 1.01
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 940.05 us/op 859.75 us/op 1.09
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3397 ms/op 2.1376 ms/op 1.09
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.4987 ms/op 1.5179 ms/op 0.99
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 4.0984 ms/op 3.6501 ms/op 1.12
Tree 40 250000 create 405.86 ms/op 293.14 ms/op 1.38
Tree 40 250000 get(125000) 197.62 ns/op 173.91 ns/op 1.14
Tree 40 250000 set(125000) 1.0163 us/op 874.91 ns/op 1.16
Tree 40 250000 toArray() 22.710 ms/op 16.057 ms/op 1.41
Tree 40 250000 iterate all - toArray() + loop 22.830 ms/op 16.104 ms/op 1.42
Tree 40 250000 iterate all - get(i) 77.124 ms/op 64.360 ms/op 1.20
MutableVector 250000 create 11.292 ms/op 10.470 ms/op 1.08
MutableVector 250000 get(125000) 7.3990 ns/op 6.2150 ns/op 1.19
MutableVector 250000 set(125000) 323.98 ns/op 251.67 ns/op 1.29
MutableVector 250000 toArray() 3.7292 ms/op 2.7625 ms/op 1.35
MutableVector 250000 iterate all - toArray() + loop 4.0180 ms/op 2.8232 ms/op 1.42
MutableVector 250000 iterate all - get(i) 1.6891 ms/op 1.4714 ms/op 1.15
Array 250000 create 3.6063 ms/op 2.7098 ms/op 1.33
Array 250000 clone - spread 1.4144 ms/op 1.0504 ms/op 1.35
Array 250000 get(125000) 0.73300 ns/op 0.49700 ns/op 1.47
Array 250000 set(125000) 0.82300 ns/op 0.57800 ns/op 1.42
Array 250000 iterate all - loop 88.984 us/op 103.71 us/op 0.86
effectiveBalanceIncrements clone Uint8Array 300000 53.293 us/op 21.653 us/op 2.46
effectiveBalanceIncrements clone MutableVector 300000 444.00 ns/op 309.00 ns/op 1.44
effectiveBalanceIncrements rw all Uint8Array 300000 175.04 us/op 159.93 us/op 1.09
effectiveBalanceIncrements rw all MutableVector 300000 129.99 ms/op 72.074 ms/op 1.80
phase0 afterProcessEpoch - 250000 vs - 7PWei 123.14 ms/op 108.46 ms/op 1.14
phase0 beforeProcessEpoch - 250000 vs - 7PWei 37.550 ms/op 40.616 ms/op 0.92
altair processEpoch - mainnet_e81889 350.49 ms/op 296.95 ms/op 1.18
mainnet_e81889 - altair beforeProcessEpoch 53.683 ms/op 47.389 ms/op 1.13
mainnet_e81889 - altair processJustificationAndFinalization 16.950 us/op 17.245 us/op 0.98
mainnet_e81889 - altair processInactivityUpdates 6.0876 ms/op 5.4973 ms/op 1.11
mainnet_e81889 - altair processRewardsAndPenalties 49.825 ms/op 69.054 ms/op 0.72
mainnet_e81889 - altair processRegistryUpdates 2.4810 us/op 2.3870 us/op 1.04
mainnet_e81889 - altair processSlashings 463.00 ns/op 527.00 ns/op 0.88
mainnet_e81889 - altair processEth1DataReset 580.00 ns/op 584.00 ns/op 0.99
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3326 ms/op 1.1749 ms/op 1.13
mainnet_e81889 - altair processSlashingsReset 5.2660 us/op 5.1960 us/op 1.01
mainnet_e81889 - altair processRandaoMixesReset 4.4030 us/op 4.5320 us/op 0.97
mainnet_e81889 - altair processHistoricalRootsUpdate 620.00 ns/op 654.00 ns/op 0.95
mainnet_e81889 - altair processParticipationFlagUpdates 2.5910 us/op 2.6250 us/op 0.99
mainnet_e81889 - altair processSyncCommitteeUpdates 748.00 ns/op 589.00 ns/op 1.27
mainnet_e81889 - altair afterProcessEpoch 140.33 ms/op 115.02 ms/op 1.22
phase0 processEpoch - mainnet_e58758 350.23 ms/op 312.11 ms/op 1.12
mainnet_e58758 - phase0 beforeProcessEpoch 107.74 ms/op 120.58 ms/op 0.89
mainnet_e58758 - phase0 processJustificationAndFinalization 15.774 us/op 15.876 us/op 0.99
mainnet_e58758 - phase0 processRewardsAndPenalties 56.354 ms/op 53.892 ms/op 1.05
mainnet_e58758 - phase0 processRegistryUpdates 8.1820 us/op 7.6260 us/op 1.07
mainnet_e58758 - phase0 processSlashings 523.00 ns/op 458.00 ns/op 1.14
mainnet_e58758 - phase0 processEth1DataReset 550.00 ns/op 499.00 ns/op 1.10
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.2822 ms/op 938.01 us/op 1.37
mainnet_e58758 - phase0 processSlashingsReset 3.6920 us/op 2.8080 us/op 1.31
mainnet_e58758 - phase0 processRandaoMixesReset 4.7520 us/op 4.4040 us/op 1.08
mainnet_e58758 - phase0 processHistoricalRootsUpdate 580.00 ns/op 532.00 ns/op 1.09
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.3480 us/op 4.1250 us/op 1.05
mainnet_e58758 - phase0 afterProcessEpoch 105.55 ms/op 95.806 ms/op 1.10
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.3313 ms/op 1.2183 ms/op 1.09
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.6175 ms/op 1.4361 ms/op 1.13
altair processInactivityUpdates - 250000 normalcase 23.702 ms/op 20.189 ms/op 1.17
altair processInactivityUpdates - 250000 worstcase 21.540 ms/op 21.956 ms/op 0.98
phase0 processRegistryUpdates - 250000 normalcase 7.7000 us/op 6.2710 us/op 1.23
phase0 processRegistryUpdates - 250000 badcase_full_deposits 272.04 us/op 241.51 us/op 1.13
phase0 processRegistryUpdates - 250000 worstcase 0.5 113.77 ms/op 125.15 ms/op 0.91
altair processRewardsAndPenalties - 250000 normalcase 53.534 ms/op 57.083 ms/op 0.94
altair processRewardsAndPenalties - 250000 worstcase 53.084 ms/op 55.108 ms/op 0.96
phase0 getAttestationDeltas - 250000 normalcase 7.3990 ms/op 6.3462 ms/op 1.17
phase0 getAttestationDeltas - 250000 worstcase 7.4476 ms/op 6.2504 ms/op 1.19
phase0 processSlashings - 250000 worstcase 3.9662 ms/op 3.4082 ms/op 1.16
altair processSyncCommitteeUpdates - 250000 190.92 ms/op 166.85 ms/op 1.14
BeaconState.hashTreeRoot - No change 286.00 ns/op 246.00 ns/op 1.16
BeaconState.hashTreeRoot - 1 full validator 55.297 us/op 48.662 us/op 1.14
BeaconState.hashTreeRoot - 32 full validator 579.27 us/op 487.65 us/op 1.19
BeaconState.hashTreeRoot - 512 full validator 5.7202 ms/op 5.0656 ms/op 1.13
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 67.499 us/op 63.003 us/op 1.07
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 974.37 us/op 864.22 us/op 1.13
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 12.966 ms/op 11.074 ms/op 1.17
BeaconState.hashTreeRoot - 1 balances 52.167 us/op 46.623 us/op 1.12
BeaconState.hashTreeRoot - 32 balances 479.73 us/op 421.96 us/op 1.14
BeaconState.hashTreeRoot - 512 balances 4.8770 ms/op 4.1720 ms/op 1.17
BeaconState.hashTreeRoot - 250000 balances 77.860 ms/op 73.056 ms/op 1.07
aggregationBits - 2048 els - zipIndexesInBitList 22.487 us/op 16.259 us/op 1.38
regular array get 100000 times 34.369 us/op 31.567 us/op 1.09
wrappedArray get 100000 times 34.698 us/op 31.541 us/op 1.10
arrayWithProxy get 100000 times 17.065 ms/op 15.440 ms/op 1.11
ssz.Root.equals 609.00 ns/op 531.00 ns/op 1.15
byteArrayEquals 623.00 ns/op 526.00 ns/op 1.18
shuffle list - 16384 els 7.5175 ms/op 6.4461 ms/op 1.17
shuffle list - 250000 els 108.89 ms/op 94.790 ms/op 1.15
processSlot - 1 slots 9.7170 us/op 8.2540 us/op 1.18
processSlot - 32 slots 1.5564 ms/op 1.2921 ms/op 1.20
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 200.85 us/op 189.52 us/op 1.06
getCommitteeAssignments - req 1 vs - 250000 vc 3.0620 ms/op 2.8704 ms/op 1.07
getCommitteeAssignments - req 100 vs - 250000 vc 4.3354 ms/op 4.1082 ms/op 1.06
getCommitteeAssignments - req 1000 vs - 250000 vc 4.5507 ms/op 4.4221 ms/op 1.03
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.0800 ns/op 4.3700 ns/op 1.16
state getBlockRootAtSlot - 250000 vs - 7PWei 583.74 ns/op 540.87 ns/op 1.08
computeProposers - vc 250000 10.774 ms/op 10.094 ms/op 1.07
computeEpochShuffling - vc 250000 106.83 ms/op 100.43 ms/op 1.06
getNextSyncCommittee - vc 250000 178.42 ms/op 167.54 ms/op 1.06

by benchmarkbot/action

@twoeths twoeths merged commit 6429685 into unstable Feb 27, 2023
@twoeths twoeths deleted the cayman/update-lightclient-bucket branch February 27, 2023 11:07
dapplion added a commit that referenced this pull request Feb 28, 2023
* v1.5.0

* Add validatior option to specify builder block selection strategy (#5064)

* Add validatior option to specify builder block selection strategy

* cleanup and improvements

* server api fx

* fix test

* fix the testcase

* Re-gossip bls changes if submitted earlier than capella (#5049)

* Cache and retransmit bls changes if submitted early

* regossip when synced on/post capella

* fix tests

* modify publish vs cache condition

* add logging

* add error logging

* batchify the processing

* fix some of the cases

* fix log issues

* also delete included keys

* fix lint

* simplify gossip loop

* fix build

* refac to a versioned bls change store

* update

* logging improvements

* improve logging and finish behavior

* cleanup

* Rename eip4844 fork as Deneb (#5082)

* Fix and extract payload to header conversion (#5101)

* Remove unused eslint disable directives (#5046)

* Add execution payload header to light client header capella onwards (#5027)

* Add execution payload header to light client header capella onwards

* add more forks

* add allfork types

* make other types multifork

* make events multifork compatible

* make lightclient routes multifork

* update light-client transport

* upgrade the blocktype on lightclient server import block

* fixes

* fix test

* fix event serialization/deserilization type

* update proof constants

* fix payload index

* update the light client update generation to multi fork

* fix types

* add spec override to pass specs

* make event types allforks

* make req/resp multifork

* make types multifork throughout

* fix the update serialize/deserialize

* fix error log

* better fork determination for update

* fix the fork of finalized

* fix tests

* fix header upgradation

* Fix zero finalized header in lightclient update gen (#5121)

* Schedule capella on sepolia (#5127)

* Get the capella builder functional (#5179)

* Allow blsChange reprocess/republish via API submission (#5185)

* Fix gossiping blsChange on submit if post capella

* comment improvement

* Use new db bucket for light client updates (#5205)

---------

Co-authored-by: Cayman <caymannava@gmail.com>
Co-authored-by: g11tech <develop@g11tech.io>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
@wemeetagain
Copy link
Member Author

🎉 This PR is included in v1.6.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants